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(57) ABSTRACT 

The present invention relates to devices and methods for the 
measurement and/or for the specification of the perceptual 
intensity of a visual image, or the perceptual distance between 
a pair of images. Grayscale test and reference images are 
processed to produce test and reference luminance images. A 
luminance filter function is convolved with the reference 
luminance image to produce a local mean luminance refer- 
ence image. Test and reference contrast images are produced 
from the local mean luminance reference image and the test 
and reference luminance images respectively, followed by 
application of a contrast sensitivity filter. The resulting 
images are combined according to mathematical prescrip- 
tions to produce a Just Noticeable Difference, JND value, 
indicative of a Spatial Standard Observer, SSO. Some 
embodiments include masking functions, window functions, 
special treatment for images lying on or near borders and 
pre-processing of test images. 

29 Claims, 4 Drawing Sheets 
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SPATIAL STANDARD OBSERVER 

This application is a continuation of prior application Ser. 
No. 1 1/045,041 filed Jan. 24, 2005 now U.S. Pat. No. 7,783, 
130. 

ORIGIN OF INVENTION 

The invention described herein was made by employees of 
the United States Government and may be manufactured and 
used by or for the Government for governmental purposes 
without payment of any royalties thereon or therefor. 

BACKGROUND OF INVENTION 

3. a Technical Field of the Invention 

This invention relates generally to the field of devices and 
methods for the specification and measurement of the percep- 
tual intensity of one or more visual images and, more particu- 
larly, to the rapid and efficient determination of a visibility 
metric for such images. 

3.b. Description of the Prior Art 

Vision is the means by which most people acquire and 
process information about the world around them. Numerous 
objects intended for human use include a component of infor- 
mation to be identified visually by a human observer. Some 
everyday examples include information displayed on a screen 
or page, keys or buttons to be pressed on a keyboard, tele- 
phone, calculator, remote control unit, among many other 
examples. Therefore, it is reasonable that the design of such 
objects include specifications to insure that the visual infor- 
mation is accessible to typical human observers, that is, that 
the information is visible. Providing a means for measuring 
and specifying visibility, a “visibility metric,” is an objective 
of the present invention. 

A significant challenge in designing standards for visibility 
is that such standards are based upon models of the human 
visual sense. However, vision is a complex and only partially 
understood process. Previous standards for visibility have 
thus tended to be complex, difficult to use and not sufficiently 
general to serve as a standard method or methods for the 
specification and measurement of visibility. The performance 
of various visibility metrics has been reviewed by Ahumada 
and coworkers in two publications: Society for Information 
Display, International Symposium, Digest of Technical 
Papers Vol. 24, pp. 305-308 (1993) and Vol. 26. pp. 45-48 
(1995), the contents of both publications are incorporated 
herein by reference. 

Other examples of visibility metrics include the work of 
Lubin and co-workers U.S. Pat. No. 6,654,504, US Patent 
Application Publication 2002/0031277 and “A Human Sys- 
tem Model for Objective Picture Quality Measurements,” 
Proceedings, International Broadcasters' Convention , 
Amsterdam, The Netherlands, pp. 498-503 (1997). These 
methods developed by Lubin and co-workers require exten- 
sive calibration for each application in addition to suffering 
from the disadvantage of complexity. These methods are 
chiefly intended for image quality evaluation. 

Other methods for estimating visibility include those of 
Barten, “The SQRI Method: A New Method for the Evalua- 
tion of Visible Resolution on a Display,” Proceedings of the 
Society for Information Display, Vol. 28, pp. 253-262 (1987). 
In addition to complexity, the Barten method suffers from the 
further disadvantage of being appropriate primarily for the 
specification of displays such as television monitors. 

Standards for the measurement and specification of color 
are known in the art and widely used. However, such color 
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standards typically do not address the spatial pattern 
employed in a visual signal (for example, the shape of a 
letter). Consequently, such methods are not appropriate for 
specifying or measuring visibility. 

5 Thus, a need exists in the art for a standard specification 
and measurement of visibility, sufficiently general to be 
applicable to large classes of visual information but suffi- 
ciently simple for widespread implementation and use, 
including embedding into inexpensive systems, 
to 

SUMMARY OF THE INVENTION 

Accordingly and advantageously, the present invention 
relates to systems and techniques for processing visual infor- 
15 mation to produce a single numerical value for the visibility 
metric indicative of a Spatial Standard Observer (SSO). 
Advantages of the SSO include a simple and efficient design 
that produces an accurate visibility metric with a relatively 
few calculations. 

20 Some embodiments of the present invention use a 
Minkowski sum directly over filtered image pixels. This tech- 
nique avoids the need for complicated spatial frequency filter 
banks, with a corresponding gain in simplicity and computa- 
tional efficiency. 

25 A particular form of Contrast Sensitivity Filter (CSF) is 
used in some embodiments of the present invention which 
combines radial- and oblique-effect filters. This permits accu- 
rate visibility predictions of the visibility of oblique patterns 
such as half-toning and rasterizing artifacts. 

30 Viewing distance and image resolution are jointly treated 
in an advantageous manner in some embodiments of the 
present invention. The use of this feature causes the computed 
value of image visibility to be substantially independent of 
image resolution (except to the extent that the resolution 
35 actually alters the visibility of the information in the image). 

A window function is advantageously employed in some 
embodiments of the present invention in such a manner as to 
represent the reduction in visibility with distance from the 
observer’s region of fixation. 

40 It is advantageous in some embodiments of the present 
invention to use convolution operations along with the win- 
dow function. In this manner it is feasible to simulate the 
scanning of an image by the eye of the observer. 

Pooling the data accumulates the visibility over the scan 
45 and is advantageously employed in some embodiments of the 
present invention. 

When images are located near a border region, it may occur 
that the border has a markedly different intensity (typically 
darker) than that of the image and the general image back- 
50 ground. In such cases, it is advantageous in some embodi- 
ments of the present invention to introduce special procedures 
for handling border effects. Two examples are presented. One 
includes at least a portion of the border into the definition of 
“image” leading to a enhanced image that is then processed 
55 by the SSO. Another approach is to attenuate the image con- 
trast near the border. 

The SSO provides a standardized measure of visibility, 
allowing comparisons to be made of visibility measurements 
taken in a wide variety of applications, locations and times. 
60 Manufacturing and engineering specifications of visibility in 
standardized units can then be made. 

Furthermore, SSO visibility measurements are not limited 
by target size. Thus, very large or very small displays can use 
SSO. 

65 The SSO further provides the feasibility of making simple, 
automated measurements of the visibility of visual informa- 
tion, not requiring the use of human observers to estimate 
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visibility. Simplicity of measurement is an important feature 
of SSO in order to allow SSO to be adopted in a wide variety 
of applications and at low cost. 

SSO has numerous potential areas of application. We note 
a few applications as illustrative of the utility of SSO, not 
thereby limiting the scope of SSO to only those enumerated. 
Many other applications are apparent to those with ordinary 
skill in the art, within the scope of the present invention. 
Possible applications include: 

Photometric Instruments incorporating SSO to produce a 
“spatial photometer” for the measurement of the visibil- 
ity of spatial patterns. 

Imaging Devices and Systems employing SSO to calculate 
the visibility of targets as viewed through those systems 
such as infrared viewing systems and remote viewing 
systems (e.g., as in unmanned aerial vehicles). 

Copier Manufacturing employing SSO to measure the vis- 
ibility of defects produced by copiers and thus test the 
copier and/or improve copier design. 

Video Codecs employing SSO in testing and/or design to 
measure the visibility of image compression artifacts 
with a view towards reducing visible defects and 
increasing bitrate. 

Display Manufacturing employing SSO to detect and mea- 
sure visible artifacts with a view towards improving and 
automating product quality control and output by only 
rejecting devices having visible artifacts. 

Graphics Software employing SSO to estimate the visibil- 
ity of graphic elements and/or to estimate the visibility 
of artifacts due to the rendering process. 

Predicting Visual Performance of Humans Following 
Vision Correction using SSO and thereby pre-evaluate 
the relative efficacy of various correction procedures 
before surgery. 

Digital Watermarking employing SSO to calculate the vis- 
ibility of a recoverable signature labeling an image that 
is intended to be invisible to a human viewer. 

These are among the advantages achieved in accordance 
with various embodiments of the present invention as 
described in detail below. 

BRIEF DESCRIPTION OF THE DRAWINGS 

To facilitate understanding, identical reference numerals 
have been used, where possible, to designate identical ele- 
ments that are common to the figures. 

The techniques of the present invention can readily be 
understood by considering the following detailed description 
in conjunction with the following drawings, in which: 

FIG. 1 depicts a high-level block diagram of a typical 
embodiment of a Spatial Standard Observer used to compute 
a visibility metric. 

FIG. 2 depicts a typical target adjacent to a dark border 
surrounding the image. 

FIG. 3 depicts a typical border aperture function that, in 
this example, has 240 columns, 180 rows, each pixel is ( Vfco ) 
degree in height and width. The value of the parameter b scale 
is 0.50 deg., and bgain is 1. 

FIG. 4 depicts a high-level block diagram of an exemplary 
computer system that can be used for implementation of 
techniques of the Spatial Standard Observer. 

DETAILED DESCRIPTION OF THE INVENTION 

After considering the following description, those skilled 
in the art will clearly realize that the teachings of the invention 
can be readily utilized for determining the probable visibility 
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of various graphical or visual depictions and displays as 
viewed by a typical human observer. In particular, the present 
invention relates generally to systems and techniques for 
processing one or more images to produce a single numerical 
5 value, or “visibility metric,” indicative of a “Spatial Standard 
Observer” (SSO). Advantages of the present invention 
include techniques for the rapid evaluation of the SSO. 

The present invention relates generally to devices and 
methods for the measurement and/or for the specification of 
10 the perceptual intensity of a visual image. Other embodi- 
ments relate generally to devices and methods for the mea- 
surement and/or for the specification of differences in per- 
ception or “perceptual distance” between two or more visual 
15 images. Such devices and methods can be advantageously 
used in situations in which it is desired to measure or to 
specify visibility or visual intensity. Examples include the 
determination of visibility and/or discriminability of text, 
graphic elements, labels, icons, among other visual images. 
20 Examples also include the determination of visibility and/or 
discriminability between images, such as an original image 
and a compressed digital form of that image. Some embodi- 
ments of the present invention can also be advantageously 
used to quantify the visibility of blemishes on a display as 
25 might be useful, for example, in providing objective determi- 
nations of pass/fail criteria in the manufacture of displays. 

In essence, various embodiments of the present invention 
operate on a digital image (or an analog image following 
digitization) or on a pair of digital images. An arbitrary num- 
30 ber of images can be compared by repeated pairwise com- 
parisons. Thus, for economy of language we will describe 
applications of the present invention to a single digital image 
or to the comparison of two digital images, understanding that 
this is by way of illustration and not limitation since multiple 
35 images can be handled by multiple applications of such pair- 
wise comparisons. Analogue images can be handled within 
the scope of the present invention following digitization by 
any of numerous digitization techniques well-known in the 
art, such as use of a digital camera, a scanner, among other 
40 devices and digitization techniques known in the field. 

In the comparison of two digital images, it is advantageous 
in some embodiments of the present invention to pre-process 
the images to erase any inessential difference before present- 
ing them as input to the SSO. Such pre-processing removal of 
45 inessential differences can improve the speed to SSO process- 
ing, further enhancing the range of potential applications 
amenable to SSO processing. 

Also, by way of illustration and not limitation, it will be 
presumed in our descriptions that the images are viewed on a 
50 particular display called the reference display, and viewed at 
a particular viewing distance. Techniques are well-known in 
the art for translating an image on a non-reference display into 
a digital representation as it would appear on the reference 
display, and for translating from an arbitrary viewing distance 
55 and angle to a standard viewing distance and angle. 

Typical inputs in the construction of a Spatial Standard 
Observer (SSO) are two digital images having (or scaled so as 
to have) the same size, called herein a test image and a 
reference image. G(x,y) is defined to be the grayscale of the 
60 pixel at column x and row y; G te Jx,y), G reference (x,y) for the 
test and reference images respectively. We take the dimension 
of the image to be n x pixels in the x direction (width) and n v 
pixels in the y direction (height). Typical values are n x =640 
and ^=480. 

65 Letting s x and s^ be the viewing angles subtended by the 
image in the x and y directions respectively, the viewing 
angles s x , s^ can be derived from the viewing distance and the 
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image size in the plane of the display by the use of Eq. 1 twice, 
once to compute s x and once to compute s . 

tan{(/r * size (degrees)/ 360} = Eq. la 

(0.5* size (cm)) /viewing distance (cm) 

360 size (cm) Eq. lb 

size (degrees) = — 

2 n viewing distance (cm) 


in which pscale is a parameter, conveniently taken to be 0.1 25 
degree in some embodiments. 

The test and reference images can be downsampled by 
integer factors in the x and y directions {d x , d v } respectively, 
5 by selecting every d x -th column and d -th row from the origi- 
nal image to create a new, downsampled image G M (x,y). This 
operation is conveniently expressed in terms of a “downsam- 
pling operator” DS as 

G\x,y)=DS{G\x,y)4 x d y ) Eq. 2.3 


Eq. lb follows from Eq. la only when the ratio (size/(viewing 
distance)) is much less than one. But this is true in virtually all 
cases of practical interest so we use Eq. lb hereinafter. Also, 
the designation of cm in Eq. la and lb is for convenience, 
since it is only necessary that “size” and “viewing distance” 15 
be expressed in the same units of length. 

The width and height of each pixel, p x and respectively, 
are given by Eq. 2 with p x , p^ in degrees if s x and s^ are in 
degrees. Typical values are s x =8 deg. and sy=6 deg. yielding 
typical values for p x =p > ,=( 1 /8o) deg. 20 


Px = — , Py = ■ 


Eq. 2 


25 


The test and reference images, G lest (x,y) andG^^foy) 
respectively, may contain noise, or may differ in those image 
components having high spatial frequencies whose visibili- 
ties are not of interest for the particular image analysis under 3Q 
consideration. In addition, the images may be captured at a 
higher resolution or larger area than is necessary for the 
particular image analysis. For these and other reasons, it may 
be useful to pre-process the test and reference images to 
remove noise, remove high frequency components and other 
components not significantly affecting the visibility analysis, 35 
to reduce image resolution, and/or to crop the image to a 
rectangle of interest (or other convenient shape). Such opera- 
tions can be performed by filtering, downsampling and crop- 
ping, pursuant to some embodiments of the present invention. 
Such operations are optional and, when employed, can be 40 
employed in any combination, sequence or number. That is, 
multiple steps of each operation can be performed whenever 
advantageous to do so, and the sequence of various operations 
or combinations can also be adjusted for the particular image 
processing task at hand. To be concrete in our description, we 45 
describe typical pre-processing operations, individually and 
in a particular sequence, understanding thereby that the 
present invention is not limited to the particular steps, 
sequence, number or type of operations described. 

It is convenient in some, embodiments to pre-filter the test 
and reference images by convolution with a pre-filter function 
PF(x,y) pursuant to Eq. 2.1 

G' (x,y)=PF(x,y) ®G(x,y) Eq. 2. 1 

for (i fes-,(x,y) and G refere „ ce {x,y) respectively. The G' function 55 
of Eq. 2. 1 , the pre-processed image, is then used in place of G 
in subsequent image processing, including in Eqs. 3, 4 and 
following. 

In some embodiments of the present invention, it is conve- 
nient to use a pre-filter function PF(x,y) given by Eq. 2.2. 


r = Y ( x Px ) 2 + (yPy? 


Eq. 2.2 
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The new dimensions of the test and reference images in the 
x and y directions are thus given as nj and nj as in Eq. 2.4. 



Eq. 2.4 


in which the function “Floor[ ]” returns the nearest integer 
less than or equal to its argument. Typical values for d^ and d y 
are d x =d y =4. 

Eq. 2.3 uses the pre-processed image G' from Eq. 2. 1 as the 
image from which the downsampled image G M is derived. 
This is a particular example presented to be concrete in our 
description and not intending to limit the scope of the present 
invention. Although downsampling is almost always pre- 
ceded by filtering to avoid aliasing, downsampling can be 
performed on an image with or without pre-filtering. 

The image G, G' or G" can be cropped to a rectangle of 
interest ROI. For definiteness, we describe cropping the G" 
image having dimensions n x T and n^V It is convenient to 
describe the ROI by the pixel coordinates of its lower left 
comer {x LL , y LL } and upper right comer {x^, y UR \ respec- 
tively. Cropping is conveniently performed by deleting from 
the image rows 1 through(y ii: -l) inclusive, and rows (y L7 ,+ l ) 
through n^' inclusive, as well as columns 1 through (x LL - 1 ) 
inclusive, and columns (x c/jR + 1 ) through n x inclusive. The 
dimensions of the new, cropped image are thus 

n x " =X UR~ X LL + 1 

n y” = yuR-yLL + 1 Eq. 2.5 

If the pre-processing techniques are used, singly or in 
combination, the resulting output images (test and reference) 
are considered the input images to the other image processing 
procedures described herein. New image dimensions (if 
present) should also be used. 

If a reference image is not readily available, it is convenient 
in some embodiments of the present invention to create one 
by deleting the target or structural component from a copy of 
the test image. If the target is confined to a local region on an 
otherwise uniform image with graylevel G 0 , then it is conve- 
nient in some embodiments of the present invention to create 
a reference image as a uniform image having the same size as 
the test image with a graylevel also equal to G 0 . Typical 
images are depicted as 1 00 in FIG. 1 with test image 1 00a and 
reference image 100/?. The structural component of the test 
image 100a is shown adjacent to the image field merely for 
clarity of depiction, understanding that the images are actu- 
ally superimposed. 

If a reference image is not available, some embodiments of 
the present invention obtain a reference image by processing 
the test image, for example, convolving the test image with a 
reference filter, RF(x,y). It is advantageous in some embodi- 
ments to pre-process the test image pursuant to one or more of 
the pre-processing techniques described herein (or others 
known in the field) before application of the reference filter, 
that is, convolve RF with G, G’, G M or equivalents, among 
others. 
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In some embodiments, it is convenient to create a reference 
image by smoothing the test image and thereby suppress from 
the test image the signals whose visibility is of interest. For 
example, smoothing can conveniently be carried out with a 
Gaussian reference filter having the form given by Eq. 2.6. 5 

RF(x, y ) = RF(r) = — ^Expf-^f — *— 
rscale 1 V V rscale 

r = a/ ( xp x ) 2 + ( yp y ) 2 


f) 


Eq. 2.6 
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“rscale” is a parameter conveniently chosen to be 2 degree. 

The reference image is then created by convolving the test 
image with the reference filter, Eq. 2.6, either by employing 15 
conventional convolution (e.g., Eq. 5a, 5b, 5c) or, advanta- 
geously according to some embodiments of the present inven- 
tion, using “confined convolution,” denoted by a “confined 

convolution operator” ,as applied in Eq. 2.7. 20 

G'"{x,y)=RF{x,y) ® c G"{x,y) Eq. 2.7 

Eq. 2.7 depicts the example in which the pre-processed 
image G"" is convolved by confined convolution to produce a 
reference image G m , understanding that pre-processing the 2 s 
test image is optional and conventional or other forms of 
convolution can be employed. 

Confined convolution offers some advantages in image 
processing. In standard cyclic convolution, the edges of the 
image are considered to be connected. Thus, image content 
close to one edge of the image may be spread over to the 
opposite edge, which is sometimes called the “wrap-around 
problem.” Confined convolution is a form of convolution 
which avoids the wrap-around problem by, in effect, discon- 
necting the opposing edges of the image. 

Confined convolution makes use of a “Pad-Convolve- 35 
Crop” (PCC) operator. The operands of the PCC operator are 
a general image function, I(x,y), and a kernel K(x,y) in which 
the kernel has k x columns and k^ rows. The image I(x,y) is 
augmented or “padded” with rows and columns containing 
entries having a value of 0, such that the padded image has k^ 40 
additional columns (of all 0’s) and k y additional rows (of all 
0’s) in comparison with I(x,y). This padded I(x,y) is con- 
volved with the kernel K(x,y). The image resulting from this 
convolution is then restored to the original image size by 
removing the added k^ rows and k x columns . This sequence of 45 
operations defines the PCC operator operating on K and I, 
denoted as PCC(K(x,y),I(x,y)). 

The confined convolution of K(x,y) with I(x,y) is then 
given by Eq. 2.8. 


K(x, y) <8 > c /(*, y) = 


PCC(K(x, y), I(x, y)) 



K(x, y) 

x^Kix, y y 



50 

Eq. 2.8 


55 


in which 1 (x,y) is an image (array) all of whose entries=l and 
which has the same number of rows and columns as the 
(unpadded) image I(x,y). 

The reference and test images (optionally, following pre- 
processing) are converted from a grayscale format to local 60 
luminance contrast image. This conversion is depicted sche- 
matically as “Contrast” 101 in FIG. 1 . The first step in this 
conversion or image transformation is the computation of a 
luminance image L(x,y) from the grayscales of each image, 
test and reference, denoted generally as G(x,y) to indicate 65 
either G teJ /x,y) or G reference (x,y) respectively. For economy 
of language we frequently omit subscripts “test” and “refer- 
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ence” using a single unsubscripted letter to indicate two equa- 
tions or two variables, one each for “test” and “reference.” 

This transformation from grayscale G(x,y) to a luminance 
image or luminance L(x,y) is advantageously performed by a 
gamma function “Gamma” as in Eq. 3. 

L (x,y) =Gamma[G(xj) ] Eq. 3 

The particular form and parameters used for the Gamma 
function will depend on the particular characteristics of the 
device displaying the test and reference images. A typical 
version is Eq. 4 in which the luminance L(x,y) is given by: 

L(x,y)=L max (G(x,y)/G max y Eq. 4 

in which L max is the maximum possible luminance in the 
image, G max is the corresponding maximum grayscale value, 
y is the gamma exponent of the display, approximately cor- 
recting for nonlinearities in the luminance characteristics of 
the display. A typical value for y is y=2.2. Eq.s 3 and 4 are 
applied to both test and reference images. 

A local luminance filter is employed having a luminance 
filter function LF(x,y). It is then convenient to introduce a 
local mean luminance reference image LL(x,y) obtained by 
the convolution of the reference luminance image k reference 
(x,y) with the luminance filter function by Eq. 5a 

LL(x,y)=LF(x,y) ®L„ /mnce (x,y) Eq. 5a 

in which ® denotes convolution of the two functions defined 
in known texts in the field, for example “Fourier Analysis and 
Imaging” by Roger N. Bracewell, (Kluwer Academic/Ple- 
num Publishers, 2003), pp. 174-179, incorporated herein by 
reference. The convolution can be expressed in discrete and 
continuous forms as in Eq. 5b and 5c respectively. 

dxdtt) Eq. 5 b 

where the integrals extend over the domain in which LF(t,oo) 
is not zero . In di screte form the convo lution i s given by Eq. 5 c . 


mx,y)®L rtfiraC '(x,y)= E 1- 5c 

llLF( Mod(* - j, n x ), Mod(y - k , n y ))L reference (j, k) 
where Mod(a, b) is the remainder when a is divided by b. 


In some embodiments of the present invention, it is conve- 
nient to use the luminance filter function LF (x,y) given by Eq, 
6 . 


LF(x, ,) = mr) = ^Exp H^f) 
r= a/ ( xp x ) 2 + (yp y ) 2 


Eq. 6 


in which lscale is a parameter to be chosen. Iflscale^+oo, this 
corresponds to an LL that is constant and equal to the average 
luminance over the image. 

The average (MEAN) luminance, L mean is given by a 
numerical average of the luminance over all pixels in the x 
and y directions, Eq. 7. 


Lmean — ^reference (A 


Eq. 7 


A typical value for L mea „ is 40 candelas per sq. meter (40 
cd/m 2 ). 
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The contrast or contrast image of each pixel, C(x,y) is then 
given by Eq. 8 applied to both test and reference luminance 
images L test (x, y) and L reference (x, y) respectively. 


C(x, y ) = 


L(x, y ) 
LL(x , y) 


Eq. 8 


loss=0.8493 

gain=373.1 

p=0.7786. 

In some embodiments of the present invention, it is conve- 
nient to choose an oblique filter, OEF having the form given 
inEq. 12. 


For the particular embodiments described thus far, L max 
plays no apparent role since it appears as a multiplicative 
factor in both L (Eq. 4), and LL (through L reference (Eq.s 4 and 
5)) hence canceling from Eq. 8. (Under the typically reason- 
able presumption that both test and reference images have the 
same maximum possible luminances, L max ). However, it is 
convenient to retain L max in the equations since it simplifies 
the application of the equations in other embodiments of the 
present invention in which l^ max and/or L mea „ may play a role 
in determining parameters of the process. A typical value for 
L max is 100 cd/m 2 . 

Following the construction of test and reference contrast 
functions via Eq. 8, both test and reference images are typi- 
cally passed through a Contrast Sensitivity Filter (CSF), 102 
in FIG. 1 . While various embodiments of CSF are feasible and 
can be used in connection with the present invention, it is 
advantageous in connection with some embodiments of the 
present invention to work in the frequency domain following 
application of a Discrete Fourier Transform, DFT, and its 
inverse DFT - 1 . In such cases, the filtering can be described by 
Eq. 9 as 


10 


15 


20 


25 


30 


F(x,y) =DFT ~ 1 [ CSF (u, v)*DFT[C(x,y )] ] Eq. 9 


in which C(x,y) is the contrast function of the image from Eq. 

8 and F(x,y) is the filtered image. 

The Discrete Fourier Transform and the Inverse Discrete 
Fourier Transform, DFT[ ] and DFT -1 [ ], are conventional 35 
operations in the field of digital signal processing and 
described in many texts, for example, the text by Bracewell, 
supra at pp. 167-168, incorporated herein by reference. 

CSF(u,v) is the discrete version of a Contrast Sensitivity 
F ilter in the frequency domain, and u and v are horizontal and 40 
vertical frequency indices respectively in units of cycles/ 
width and cycles/height. 

The discrete, frequency domain version of the Contrast 
Sensitivity filter, CSF(u,v) is conveniently given by the prod- 
uct of a radial contrast sensitivity function, RCSF(u,v), and an 45 
oblique effect contrast sensitivity filter, OEF(u,v), as 
expressed in Eq. 10. 


CSF(u,v)=RCSF(u, v)OEF{u,v) Eq. 10 

In some embodiments of the present invention it is conve- 
nient to choose a radial function RCSF having the form given 
in Eq. 1 1 . 


OEF(u, v) = OEF(f, 6) Eq. 12 

-'-('-M-tIF)) 

Sin 2 (20) if / > corner 
= 1 if / < corner 



in which “comer” and “slope” are parameters. Typical values 
for “comer” and “slope” are corner=3.481 and 
slope=l 3.571 49. 

Following processing of both the test image and the refer- 
ence image by CSF, 102, the resulting filtered images are 
subtracted pixel -by -pixel, 103. The result is the difference 
image D(x,y) of Eq. 13. 


D{x,y)=F test {x,y)-F reference {x,y) Eq. 13 

In some embodiments of the present invention, it is advan- 
tageous to create a mask image, M(x,y), from the filtered 
reference image F re y ferewce (x,y). In such embodiments, the 
absolute value of the filtered reference image is raised to a 
power “a ”, convolved with a masking filter MF(x,y), added to 
the constant 1 and the b’th root of the resulting expression is 
computed as in Eq. 14. 


M(x, y) = [1 + MF(x, y) ®\F reference 


(x, y)\ a ]b 


Eq. 14 


in which the convolution operator Vindicates discrete con- 
volution. 

In some embodiments, it is advantageous to choose a=b=2 
in Eq. 14, resulting in a mask image M(x,y) given by Eq. 15. 

M(x,y)=]j 1 +MF{x,y) ®F^ nc 2 (x,y) Eq. 15 

Furthermore, it is advantageous in some embodiments of 
the present invention to choose the masking filter MF(x,y) to 
have the form of Eq. 16 


MF{ x, .y) = MF(r) = mgairiExp[-n ^^^ j j 


Eq. 16 


RCSF(u, v) = RCSF(f) Eq. 11 55 

= gain sech^y- j j - loss sech^y- j 



in which “mgain” and “mscale” are parameters. Typical 
choices for mgain and mscale are mgain=0.2 and mscale=0. 1 . 

In some embodiments of the present invention, the differ- 
ence image D(x,y) is divided by the masking image to yield a 
masked difference image MD(x,y) according to Eq. 17. 


in which “sech” is the hyperbolic secant function, “gain”, MD(x, y) = ^ ^ 

“loss”, f 0 , f x and p are parameters. Typical values for these 
parameters are as follows: 65 

f 0 =4.173 
f x =1.362 


For those embodiments in which a mask image is not 
employed, the masked difference image is simply the differ- 
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ence image. Also, when a mask image is not employed, the 
subtract operation 103 can optionally precede the CSF 102. 

At this “boost” stage, 106 in FIG. 1, the absolute value of 
the masked difference image is computed, raised to a power (I, 
and convolved with a window function W(x,y). The result of 5 
these operations is a function that is then raised to the power 
1/p and multiplied by the factor (p x p^) 1/p to produce a Just 
Noticeable Difference Image JND(x,y) as in Eq. 1 8. A typical 
value for p is p=2.408 


1 4 Eq. 18 

JND(x, y ) = (p x p y )P[W(x, y)®\MD(x , y)f] 

In some embodiments, it is advantageous to use a window 
function W(x,y) as given by Eq. 19 


in which “wscale” is a parameter, advantageously chosen to 
be approximately 1.013 in some embodiments. 

It is advantageous in some embodiments of the present 25 
invention to display the complete JND(x,y) image, 107 in 
FIG. 1. While optional, such a display can provide a useful 
visual indication of the location and magnitude of visual 
signals. The JND(x,y) image can be “thresholded” (setting to 
zero values less than a threshold, T), reduced in size (or 30 
otherwise scaled), and/or converted to a portable format to 
provide a compact record of the information. 

The next stage in the process combines or “pools” the 
values of JND(x,y) of the pixels in the x and y directions to 
produce a single value of JND. It is convenient to use a 35 
Minkowski summation to effect this pooling with a parameter 
i|) as exponent, as given in Eq. 20. 


jnd = {p x p y )* 


1 JND ^,y)f 


Eq. 20 40 


The number, JND of Eq. 20 is the desired numerical value 45 
characterizing the Spatial Standard Observer. 

In some embodiments, it is advantageous to let co— >oo 5 in 
which case Eq. 20 reduces to Eq. 21. 

JND=Mslx [JND( x,y ) ] Eq. 21 5Q 

In some embodiments of the present invention, it is advan- 
tageous to apply a non-linear transformation (for example, a 
power function) to the JND computed from either Eq. 20 or 
Eq. 21. Thus, whether or not a non-linear transformation is 
applied to JND, and whether or not border effects are relevant 55 
for the particular image(s) under consideration, the Spatial 
Standard Observer, as characterized by the value of JND, 
provides an effective visibility metric, able to be computed 
relatively rapidly. 

In some applications, the target or test image (201 in FIG. 60 
2) may be located adj acent to a border 2 00 of the image region 
202, as depicted in FIG. 2. If the region of the display outside 
the image, 200, is darker than the display, 202, for example, 
the dark region of a Liquid Crystal Display (LCD) panel, then 
the visibility of the target 201 in the region will typically be 65 
reduced. An example of this situation is depicted in FIG. 2. 
Thus, it is advantageous in some embodiments of the present 


invention to use special techniques for the treatment of border 
areas in order to produce correct visibility estimates for such 
targets. 

In some embodiments of the present invention it is advan- 
tageous to multiply the contrast images by a spatial border 
aperture function BA(x,y) between the Contrast and CSF 
steps, that is, at 120 in the process flow diagram of FIG. 1 . The 
resulting Contrast Border Aperture Function, CBA(x,y) is 
thus 


CBA (x,y)=C(x,y)BA (x,y) Eq. 22 

Then CBA(x,y) is used in place of C(x,y) at the CSF step, Eq. 
9. 

In some embodiments of the present invention, the border 
aperture function is advantageously chosen to be: 


(x-l )p x , ] 2 ) 


Min 


(y-i)p y , 

(n x -x)p x . 


BA(x, }>)=!- bgainExpl 


—71 


K - y)p y 

bscale 2 


Eq. 23 


in which “bgain” and “bscale” are parameters. An example of 
this function is given in FIG. 3 in which the image is taken to 
have 240 columns (x-coordinate) and 180 rows (y-coordi- 
nate). Each pixel in this example is taken to be (%o) degree in 
both height and width. The parameters are chosen in this 
example as bscale=0.5 degree and bgain=l. 

The use of a border aperture function, BA(x,y) as in Eq. 23, 
has the advantage of simplicity, but as an approximation, it 
may not be as accurate as alternative methods. In other 
embodiments, it is advantageous for the parameters bscale 
and bgain to depend upon the luminance contrast between the 
image and the border. Typically, a margin is added to the 
image such that the enlarged image, image+margin, contains 
a portion of the border. This enlarged image is then processed 
as the “image” pursuant to the image processing techniques 
described herein, typically including the masking component 
of the processing, 105 . The presence of a portion of the border 
in the enlarged image will tend to produce the appropriate 
masking effect, tending to reduce visibility of targets or por- 
tions of targets near the border. 

There are various ways the use of an enlarged image can be 
implemented to treat border effects. For example, it is conve- 
nient to take the width of the border region to be Round 
[2*mscale/pJ and the height to be Round[2*mscale/p > ], in 
which mscale is the masking parameter (Eq. 1 6). “Round[ ]” 
is a function that generates as the value of the function that 
integer nearest to the value of the function’s argument. The 
dimensions of the enlarged image are then given by Eq. 24 as: 

width=K JC +Rouiid[2 *mscale//> x ] 


height=^+Round[2 ^mscale/p^] Eq. 24 

An advantage of treating border effects with an enlarged 
image is that it more correctly deals with the dependence of 
the border masking effect upon the luminance contrast 
between the border and the (original, unenlarged) image. A 
possible disadvantage is that this approach requires some- 
what more processing to include the masking step. 

JND from Eq. 20 (or Eq. 21 for fl^oo) relates to the 
percentage of human observers who will notice a difference. 
For example, images leading to JND having a value around 1 
will typically present noticeable differences to about 75% of 
typical human observers. Images resulting in larger JND val- 
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ues will present noticeable difference to a correspondingly 
larger percentage of typical human observers, although the 
precise functional relationship between JND and the percent- 
age of viewers observing differences may not be readily 
known. 

ft is advantageous in some embodiments of the present 
invention to use JND as a measure of different levels of 
perceptual intensity. That is, larger JND values indicate that a 
larger percentage of observers will notice a difference. But 
also larger values of JND typically indicate that a given 
observer will be more likely to observe more detailed differ- 
ences. By way of illustration and not limitation, we consider 
the example of observing a scene through some form of 
optical instrument, such as a remote viewing device, night 
vision goggles, among others. A given observer may require 
an image value of JND X in order to conclude that some object 
is present other than natural background. However a value of 
JND 2 >JND 1 would be required for the observer to conclude 
that the object is a military vehicle. And a value of 
JND 3 >JND 2 would be required to conclude that it is a hostile 
military vehicle. Thus JND values as determined by the SSO 
can be a useful measure of not only minimal levels of visibil- 
ity but, when more stringently applied, also estimate the 
probable level of perceptual information obtainable from a 
given image. 

FIG. 4 depicts an illustrative computer system 250 that 
utilizes the teachings of the present invention. The computer 
system 250 comprises a processor 252, a display 254, input 
interfaces 256, communications interface 258, memory 260, 
and output interfaces 262, all conventionally coupled by one 
or more busses 264. The input interfaces 256 comprise a 
keyboard 266, mouse, trackball or similar device 268, as well 
as mass- storage input devices such as CDs, DVDs, magnetic 
discs of various designs among others. The output interface 
262 is a printer 272. The communications interface 258 is a 
network interface card (NIC) that allows the computer 250 to 
communicate via a network, such as the Internet. Image 
acquisition/generation devices 274 provide the images 100 
for the generation of the SSO and are also coupled to the 
processor 252. The units 274 can supply either stored or 
realtime input data, or both. 

The memory 260 typically comprises different modalities, 
illustratively semiconductor memory, such as random access 
memory (RAM), and disk drives. Depending on the embodi- 
ment, the memory 260 typically includes an operating sys- 
tem, 280. The operating system 280 may be implemented by 
any conventional operating system such as UNIX®, WIN- 
DOWS®, and LINUX®, among others. 

Although various embodiments which incorporate the 
teachings of the present invention have been shown and 
described in detail herein, those skilled in the art can readily 
devise many other varied embodiments that still incorporate 
these teachings. 

What is claimed is: 

1. A method of processing an image, the method compris- 
ing: 

producing a test image; 

producing a test luminance image from the test image; 

producing a reference image; 

producing a reference luminance image from the reference 
image; 

producing a local mean luminance reference image as a 
convolution of the reference luminance image and a 
luminance filter function; 

producing a test contrast image in the absence of temporal 
filtering; 

producing a reference contrast image; 
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producing a difference image; and 
producing a just noticeable difference image as a math- 
ematical combination of the difference image, 
wherein the convolution is defined as confined convolu- 
tion, which comprises: 
receiving an image; 

padding the image with zeros to provide a first intermediate 
image; 

convolving the first intermediate image with a selected 
non-negative kernel function to obtain a second inter- 
mediate image; 

cropping the second intermediate image to obtain a third 
intermediate image; 

receiving said third intermediate image, I3(x,y)=PCC{K 
(x,y),I(x,y)}; and 

forming a fourth intermediate image, 
defined as I4(x,y)=K(x,y 0 t7 1(x,y)=PCC{K(x,y), I(x,y)}/ 
PCC {K(x,y)/2x2yK(x,y),I(x,y)}. 

2. A method of spatially processing an image, the method 
comprising: 

spatially producing a test image with a test image dimen- 
sion of n x pixels in the x direction (width) and n^, pixels 
in the y direction (height) having G test (x,y) which is 
defined to be the grayscale of the pixel at column x and 
row y; 

spatially producing a reference image with a reference 
image dimension of n^ pixels in the x direction (width) 
and n y pixels in the y direction (height) having G reference 
(x,y) which is defined to be the grayscale of the pixel at 
column x and row y; 

wherein spatially producing the test and reference images 
includes: 

providing viewing angles subtended in each image in the x 
and y directions defined by s x and s y respectively, the 
viewing angles s x , s^, can be derived from a viewing 
distance and an image size in a display by the equation as 
follows, once to compute s x and once to compute s^: 

tan {(jt*size(degrees)/360}=(0.5*size)/viewing dis- 
tance 

and 

providing a width and height for each pixel, p x and p^ as 
follows: 



50 producing a test contrast image; 

producing a reference contrast image; 
producing a difference image; and 
producing a just noticeable difference image as a math- 
ematical combination of the difference image. 

55 3. The method of claim 2, wherein a reference luminance 

image is produced from the reference image and a test lumi- 
nance image is produced from the test image. 

4. The method of claim 3, wherein a local mean luminance 
reference image is produced as a convolution of the reference 

60 luminance image and a luminance filter function. 

5. The method of claim 4, wherein the test contrast image 
is produced by a mathematical combination of the test lumi- 
nance image and the local mean luminance reference image. 

6. The method of claim 4, wherein the test contrast image 

65 is produced by a mathematical combination of the test lumi- 
nance image, the local mean luminance reference image and 
a border aperture function. 
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7. The method of claim 4, wherein the test contrast image 
is produced by a mathematical combination of a test lumi- 
nance image, the local mean luminance reference image and 
an image of a border surrounding the reference image. 

8. The method of claim 4, wherein the reference contrast 5 
image is produced by a mathematical combination of the 
reference luminance image and the local mean luminance 
reference image. 

9. The method of claim 4, wherein the reference contrast 
image is produced by a mathematical combination of the 
reference luminance image, the local mean luminance refer- 
ence image and a border aperture function. 

10. The method of claim 2, wherein the just noticeable 
difference image is produced as a mathematical combination 15 
of the difference image with a window function. 

11. The method of claim 10, wherein the window function 
is convolved with the difference image. 

12. The method of claim 2, wherein the difference image is 
produced by subtracting the reference image from the test 20 
image to produce the difference image. 

13. The method of claim 2, wherein a contrast sensitivity 
filter is applied to the test contrast image to produce a filtered 
test image. 

14 . The method of claim 2, wherein the contrast sensitivity 25 
filter is applied to the reference contrast image to produce a 
filtered reference image. 

15. The method of claim 12, wherein the difference image 
is produced by subtracting the filtered reference image from 
the filtered test image to produce the difference image. 

16. The method of claim 2, wherein the test contrast image 
is produced by producing a mask image as a mathematical 
combination of the reference image with a masking filter, and 
producing a difference image as a ratio of the difference 35 
image and the mask image. 

17. The method of claim 14, wherein the test contrast 
image is produced by producing a mask image as a math- 
ematical combination of the filtered reference image with a 
masking filter, and producing a difference image as a ratio of 40 
the difference image and the mask image. 

18. The method of claim 2, wherein a visibility metric is 
produced by pooling the just noticeable difference image. 

19. The method of claim 18, wherein the process of pooling 
combines the values of the pixels in the x and y directions of 45 
the just noticeable difference image to produce a single just 
noticeable difference value. 

20. The method of claim 2, further comprising preprocess- 

ing at least one of the test image and the reference image by 
downsampling. 50 

21. The method of claim 2, further comprising preprocess- 
ing at least one of the test image and the reference image by 
convolution with a selected pre-filtering function. 

22. The method of claim 2, further comprising preprocess- 
ing, by cropping, at least one of the test image and the refer- 55 
ence image. 

23. The method of claim 2, further comprising: 

the test image having first and second opposing sides; and 

performing a convolution of the test image with a selected 
filter that isolates the first and second opposing sides of 60 
the test image from each other, to thereby form the 
reference image. 

24. A method of performing confined convolution, the 
method comprising: 

receiving an image; 65 

padding the image with zeros to provide a first intermediate 
image; 
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convolving the first intermediate image with a selected 
non-negative kernel function to obtain a second inter- 
mediate image; 

cropping the second intermediate image to obtain a third 
intermediate image; 

receiving said third intermediate image, I3(x,y)=PCC{K 
(x,y),I(x,y)}; and 

forming a fourth intermediate image, 

defined as I4(x,y)=K(x,y ® c I(x,y)=PCC{K(x,y), I(x,y)}/ 
PCC {K(x,y)/2x2yK(x,y),I(x,y)}. 

25. A method of processing a spatial image, the method 
comprising: 

producing a spatial test image with a test image dimension 
of n x pixels in the x direction (width) and n y pixels in the 
y direction (height) having G test (x,y) which is defined to 
be the grayscale of the pixel at column x and row y; 
producing a spatial reference image from the spatial test 
image with a reference image dimension of n x pixels 
in the x direction (width) and n r pixels in the y direc- 
tion (height) having G reference (x,y) which is defined to 
be the grayscale of the pixel at column x and row y; 
wherein spatially producing the test and reference 
images includes: 

providing viewing angles subtended in each image in the 
x and y directions defined by s x and s >; respectively, the 
viewing angles s x , s^ can be derived from a viewing 
distance and an image size in a display by the equation 
as follows, once to compute s x and once to compute s^: 

tan {(jt*size(degrees)/360}=(0.5*size)/viewing dis- 
tance 

and 

providing a width and height for each pixel, p x and p y as 
follows: 



producing a test contrast image; 
producing a reference contrast image; 
producing a difference image; and 

producing a just noticeable difference image as a math- 
ematical combination of the difference image. 

26. A method of spatially processing an image, the method 
comprising: 

producing a spatial test image with a test image dimension 
of n x pixels in the x direction (width) and n v pixels in the 
y direction (height) having G test (x,y) which is defined to 
be the grayscale of the pixel at column x and row y; 
producing a spatial reference image with a reference 
image dimension of n x pixels in the x direction (width) 
and n v pixels in the y direction (height) having 
G re f er ence (x,y) which is defined to be the grayscale of 
the pixel at column x and row y; 
wherein spatially producing the test and reference 
images includes: 

providing viewing angles subtended in each image in the 
x and y directions defined by s x and s^ respectively, the 
viewing angles s x , s^ can be derived from a viewing 
distance and an image size in a display by the equation 
as follows, once to compute s x and once to compute s v : 

tan {(jtsize(degrees)/360}=(0.5*size)/viewing dis- 
tance 
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and 

providing a width and height for each pixel, p x and p y 
follows: 


$x _ Sy . 


producing a test contrast image; 
producing a reference contrast image; 
producing a difference image; and 
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producing a just noticeable difference image as a math- 
ematical combination of the difference image with a 
window function. 

27. The method of claim 26, wherein the window function 
5 is convolved with the difference image. 

28. The method of claim 27, wherein the convolution is 
defined as confined convolution. 

29. The method of claim 2, wherein the test contrast image 
is produced in the absence of temporal filtering. 

10 



